Repository navigation
feat(evals): add opt-in per-slice quality gates - #755
Conversation
|
Thanks @Bruce-Yii for the thorough review. I pushed 136ae2a to address the follow-ups:
Local verification: 98 eval tests, 25 eval API checks, Ruff, compileall, and the strict docs build. On this head, GitHub reports 24 successful checks and one skipped deployment check. The separate SDK link fix overlaps #742. Since #742 is still open and the current strict docs build needs the correction, I will drop the overlapping commit once #742 lands. Maintainer help needed: |
Bruce-Yii
left a comment
There was a problem hiding this comment.
This holds up under adversarial probing, which is the part I expected a quality gate to get wrong. I threw 30 false-green vectors at it and every one fails closed with a specific message:
- A slice regressing while the overall metric improves. Baseline en n=180 @0.90 + zh n=20 @0.90, candidate en 0.95 + zh 0.60 — overall goes 0.9000 → 0.9150. The legacy gate passes it (exit 0); with
--gate-policyit fails, and the message names the numbers:candidate=0.6000 (n=20), baseline=0.9000 (n=20), max_drop=0.0500 exceeded by 0.2500. That is the whole point of the feature and it works. - Insufficient evidence on either side. A baseline whose aggregate still reads 0.9000 while
casesholds 19 rows fails onbaseline count 19 is below min_count 20— the count is derived from the cases that actually carry a finite score, not from the stored aggregate. Removing either count guard turns a test red. - Skipped rows, including where the count check cannot be what fires.
run --on-error skipwhere one case raises leaves zh at n=9; under a loosemax_drop 0.9that the numbers would happily pass,comparestill exits 1 withslice gate: candidate has skipped/errored cases. This works becauseevals._identity_ofcopies the whole config rather than just the identity keys, soerroredsurvives the compare filter. - Order independence. Three rules, all six permutations, one distinct verdict. The loop has no
breakand no state carries between rules, so there is nothing for order to change. - Also: slice present on one side only, unknown metric / slice value / dimension / policy version,
NaNandInfinityandtrue-as-1in limit fields, zero-count and empty groups, non-comparable and identity-less baselines,ecedrift on a lower-is-better metric, a baseline already below the floor, and reports with missing or malformedcases.
I could not find a path through check_policy that returns an empty failure list for a violated rule. Exit codes match the repo convention (0 / 1 / 2) across 24 observed cases, and a report round-trips: config.gate_policy survives load/dump(sort_keys=True), and reloading through compare with the same file reproduces the identical exit code and message. cases is what makes that work, so it surviving serialization is load-bearing.
Corroborating this independently: 11 of 19 mutations of the new code go red, including both load-bearing wirings (run not applying the policy, compare not applying the policy), the pre-checkpoint validation, and the identity guard. The no-flag path is byte-identical to main across 11 invocations — plain run, run with baseline, run with thresholds, a failing run, markdown output, compare both ways, validate, two usage errors, and both report JSONs. All 24 checks on this head are green, including docs (strict build).
Not blockers, but four things I would fix
- A report records the policy it ran under, and
comparenever reads it back. Run with{min: 0.99}on a regressed candidate exits 1 and stamps that strict policy into the report. Thencompare report.json --gate-policy {min: 0.0}exits 0, with nothing indicating the artifact was produced under a stricter gate. The applied verdict is still computed from the policy named on the command line, so this is a reporting gap rather than a wrong answer — butevals_cli.py:223-235argues that "a gate that cannot say which experiment produced a number cannot support a promotion decision", and this is the remaining hole in that. Oneifcomparingconfig.gate_policyagainst the applied policy would close it. test_slice_gate_counts_each_tagged_answer_once_and_accepts_boundarydoes not test the first half of its name. The assertion ismin_count: 200over 200 cases; inflating the count 3× still satisfies>= 200._countuses tag membership rather than iteration and is correct today, but nothing would catch a regression.- The slice-dimension whitelist in
load_policy:48is untested. The malformed-policy cases cover two-limits, two-dimensions,min_count: 0and negativemax_drop, never an unknown dimension name. _eval_policy.py:150-151,if base_report is None: continue, is unreachable — any relative rule setsneeds_baseline, which returns at:122-124first. It is also the only place in the rule loop that skips a rule without recording a failure, so it is fail-open-shaped code that is currently harmless. Deleting it removes the chance of a future refactor turning it into a real bypass.
On the bundled docs fix
The docs/typescript-sdk.md hunk here is byte-equivalent to #742's: same removed line, same added URL, same result. Once #742 lands, your hunk becomes a no-op; if #752's rewrap wins instead, your one-line version conflicts with it. Worth dropping commit 5656848 and rebasing — that is all it is, and the Docs workflow runs for laya/ changes either way, so bundling it does not buy you a green check any sooner.
Two smaller notes: config.gate_policy is documented in docs/evals.md:176 but is not in the derived key list at tests/test_evals.py:943-948, whose stated contract is that it is derived from the code and therefore cannot drift. And _count requires isinstance(correct, bool) for ece while evals.py:488 accepts any non-None, so a third-party report storing correct: 1/0 as ints fails every ece rule on min_count — fail-closed and consistent with "missing evidence never passes", just undocumented.
Not verified: I ran no real checkpoint or Router-backed evaluation, so a real router's model slice values interacting with a policy is untested end to end. mergeable_state=blocked here is a stale tests (py3.9) required check — ci.yml's matrix is 3.10–3.13 and no run on this SHA produces that job. That is a repo-settings matter, not a change to make in this PR, and your note asking about it is the right call.
Bruce-Yii
left a comment
There was a problem hiding this comment.
Re-reviewed the current head after your follow-ups. The policy-mismatch path now behaves as intended: a saved report that records one gate policy and is compared with another produces a warning, while reports with no recorded policy remain intentionally silent. The duplicate-tag boundary, unknown-dimension validation, unreachable skip, and report-key follow-ups are also addressed.
I reran the changed behavior and did not find a new false-green path in the areas touched by this update. APPROVE on 136ae2a54796ece76eadc917434f9eff9e305388.
The overlapping SDK-link hunk can still drop once #742 lands; I do not consider that a blocker here.
|
Merged for 0.3.23. The adversarial pass on this is what earned it: a slice regressing while the overall metric improves is the exact false green a quality gate exists to catch, and failing closed on missing evidence, skipped cases and unsuitable relative baselines is the part most implementations get wrong. The policy-mismatch warning on The overlapping SDK-link hunk dropped out cleanly, since #742 came in via #684 in the same pass. Verified on the integrated branch: Thank you. |
What
language,model,qid, andtagslices. Each rule names an exact metric, a minimum scored count, and either an absolute bound or a directional baseline drift limit.laya-evals runand saved-reportcompare; record the run policy in the JSON report. Missing evidence, skipped cases, and unsuitable relative baselines fail closed. Existing commands without the flag keep their behavior.Why
The current baseline gate compares
overallonly. In the regression test, English improves from 90% to 95% across 180 answers while Chinese falls from 90% to 60% across 20. Overall rises from 90% to 91.5%, so an overall tolerance of 0.02 passes. Alanguage=zhrule withmax_drop: 0.05catches the 0.30 drop. This makes a reviewed critical slice enforceable in CI without changing the default comparison.How it was verified
python -m pytest tests/test_evals.py -q -k "slice_gate or slice_policy or slice_rule or invalid_policy_json"— 12 passed locally.python tests/test_evals_api.py— 25 passed locally.ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120andpython -m compileall -q laya/ tests/— passed locally.zensical build --strict --clean— passed locally with no warnings after the SDK link correction.The complete eval suite could not run locally because the CPU
torchdownload was interrupted; the repository CI installed it and ran the suite successfully.Checklist
mainFixes #754